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Abstract 



In a sample of 19 million produced B mesons, we have ob- 
served the decays B rjK* and improved our previous measurements 
of i? — > ij'K. The branching fractions we measure for these decay 
modes are B{B+ r]K*+) = (26.4;|^ ± 3.3) x 10-^ B{B^ r]K*^) = 
(13.8t|| ± 1.6) X 10-^ B{B+ r]'K+) = (80t^° ± 7) x 10"*^ and 
B{B^ r]'K^) = (89lig ±9) x 10"*^. We have searched with comparable 
sensitivity for related decays and report upper limits for these branching frac- 
tions. 
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There has been considerable recent interest in charmless hadronic B decays, partly be- 
cause of the observation of several of these decays [0-0], and partly because of their an- 
ticipated importance in understanding the phenomenon of CP violation. These decays are 
expected to proceed primarily through b ^ s loop ( "penguin" ) diagrams and h ^ u specta- 
tor diagrams. In Fig. ^ we show four such diagrams which may be expected to contribute 
to the decays involving isoscalar mesons which are the subject of this Letter. An earlier 
search found a large rate for the decay B f]'K, and set upper limits on other decays 
to two-body final states containing t] or rj' mesons. Recent effective Hamiltonian predictions 
IQJ^ of the decay branching fractions of the B rj'K decay are still somewhat smaller than 
the measurement [Q. 



3300198-001 




(c) (d) 

FIG. 1. Feynman diagrams describing the representative decays — > r]^''> K^*'^^ : (a, b) 
internal penguins; (c) external spectator; (d) flavor-singlet penguin. 

We present results of improved experimental searches for B meson decays to two-body 
final states containing rj or rj' mesons with the first observation of the decay B r]K*. These 
results are based on data collected with the CLEO II detector at the Cornell Electron 
Storage Ring (CESR). The data sample corresponds to an integrated luminosity of 9.13 fb~^ 
for the reaction e+e" — > T(4S') — > BB, which in turn corresponds to 9.66 x 10^ BB pairs 
101 . To study background from continuum processes, we also collected 4.35 fb~^ of data at a 
center-of-mass energy below the threshold for BB production. These constitute the complete 
data sample from the CLEO II and CLEO II. V experiments, and the measurements we report 
here supersede our earlier results [0] from a subset of these data. 

The CLEO II detector emphasizes precision charged particle tracking, with specific ion- 
ization (dE/dx) measurement, and high resolution electromagnetic calorimetry based on 
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CsI(Tl). Scintillators between the tracking chambers and calorimeter provide time-of-flight 
(TOF) information which we use in conjunction with dE/dx for particle identification (PID). 
The CLEO II. V configuration differs in two respects: the replacement of an inner straw- 
tube drift chamber with a three-layer, double-sided-silicon vertex detector; and replacement 
of the 50:50 argon-ethane gas in the main drift chamber with a 60:40 helium-propane mixture. 

We reconstruct charged pions and kaons, photons, and tt+tt^ pairs that intersect at a 
vertex displaced from the collision point ("vees" from K'g tt+tt^). Candidate B decay 
tracks must meet specifications on the number of drift chamber measurements, goodness of 
fit, and consistency with an origin at the primary or particular secondary vertex. Candidate 
photons (from vr", r/, and 7]' decays) must be isolated calorimeter clusters with a photon-like 
spatial distribution and energy deposition exceeding 30 MeV. In order to reject soft photon 
backgrounds, we require — > 77 candidates to satisfy |cos^*| < 0.97, where 9* is the center- 
of-mass decay angle relative to its flight direction. This cut is tightened to 0.90 for rjK* / p 
channels to veto B K*'y background. We reject charged tracks and photon pairs having 
momentum less than 100 MeV/c. The photon from candidate r]' — p7 decays is required to 
have an energy greater than 200 MeV, though this requirement is relaxed to 100 MeV for 
channels with relatively low background. 

We fit photon pairs and vees kinematically to the appropriate combined mass hypothesis 
to obtain meson momenta. The reconstructed mass resolutions prior to the constraint are 
about 5-10 MeV (momentum dependent) for 7r° 77, 12 MeV for t] — > 77, and 3 MeV for 
Kg TT+TT^. We determine the expected signal distributions for these and other quantities 
needed in the analysis from a detailed geant based simulation of the CLEO detector and 
studies of data for a variety of benchmark processes. In particular, we have determined the 
momentum and dE/dx resolutions in studies of K~tt^ data events for track momenta 

greater than 2.0 GeV/c. 

The primary means of identification of B meson candidates is through their measured 
mass and energy. The quantity AE is defined as AE = Ei + E2 — Ef,, where Ei and E2 are 
the energies of the two B daughters (typically ~2.6 GeV) and Et is the beam energy (5.29 
GeV). The beam-constrained mass of the candidate is defined as M = \J El — |pp, where p 
is the measured momentum of the candidate (typically |p| ~ 325 MeV/c). We use the beam 
energy instead of the measured energy of the B candidate to improve the mass resolution 
by about one order of magnitude. 

For vector-pseudoscalar decays of the B and the p7 decay of the 77', we gain further 
discrimination from the helicity variable Ti, the cosine of the vector meson's rest frame two- 
body decay angle with respect to its flight direction, which reflects the spin alignment in 
the decay. The decay rate is proportional to Ti^ when the vector meson decays into two 
spinless daughters, and to 1 — for rj' — > p7. For modes in which one daughter is a single 
charged track, dE/dx measurements provide statistical discrimination between kaons and 
pions. With Sk and S*^ defined as the deviations from nominal energy loss for the indicated 
particle hypotheses measured in standard deviations, the separation Sk — S-,, is about 1.7 
(2.0) at 2.6 GeV/c for the CLEO II (II. V) samples. 

The large background from continuum quark-antiquark (gg) production can be reduced 
with event shape cuts. Because B mesons are produced almost at rest, the decay products 
of the BB pair tend to be isotropically distributed, while particles from qq production have 
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a more jet-like distribution. The angle 6t between the thrust axis of the charged particles 
and photons forming the candidate B and the thrust axis of the remainder of the event is 
required to satisfy | cos 6t\ < 0.9. Continuum background is strongly peaked near 1 and signal 
is approximately flat for | cos 6t \ ■ We also form a multivariate discriminant (JF) [|l^ from 
the momentum scalar sum of charged particles and photons in nine cones of increasing polar 
angle around the thrust axis of the candidate, the angle of the thrust axis of the candidate, 
and the direction of p with respect to the beam axis. We have checked the backgrounds from 
the dominant B decay modes {b — >■ c) by simulation, finding their contributions to the modes 
in this study to be generally quite small. Where appropriate we include this component in 
the fits described below. 

The selection criteria for mass, energy, and event shape variables are chosen to include 
sidebands about the expected signal peaks. To extract event yields we perform unbinned 
extended maximum likelihood fits to the data [^]. Observables for each event include M, 
AE, JF, and (where applicable) resonance masses and Ti. The number of events included in 
the fits ranges from ~100 to 20,000. 

For B^ decays |TT| that have a primary daughter charged hadron (generically h^) that 
can be either vr"*" or K~^, we fit both modes simultaneously, with the likelihood C expanded 
so that the signal and background yields of both tt"'" and are fit variables. The modes 
with a secondary vector decay involving /i+ {K* — *• K~^7i and p — > tt+tt) also require special 
treatment. For these modes the momentum spectrum of is bimodal because of the 
forward-backward peaked Ti distribution. We select independent K* and p samples to fit 
according to the sign of Ti. Events with H < in our sign convention have low momentum 
h'^ and are unambiguously separated by kinematics combined with PID information from 
dE/dx measurements. For the events with 7i > the separation is much smaller, so we fit 
both K* and p yields simultaneously, using the K* hypothesis for Ti.. In all cases involving 
two hypotheses, we include the normalized dE/dx observables S'^r and Sk in the fit. For 
K*^ , we distinguish if+vr" from K~tt^ candidates using dE/dx and TOF information. The 
kinematics and the definition of Ti for these neutral decays causes ~85% of all p° 7r~^n~ 
signal candidates to be assigned to the Ti. > sample. All possible combinations are included 
except K*^ K^tt^ (efficiency too small) and 7i > with low momentum tt*^ for the rj' p7 
channel (background too large). 

The probability distribution functions (PDF) are constructed as products of functions 
of the observables. For signal the dependences on masses and energies are represented by 
Gaussian, double Gaussian, or Breit-Wigner functions, whose parameters are fixed in the 
fit. The background PDF contains signal-like peaking components in its resonance mass 
projections, to account for real resonances in the background, added to smooth components 
for combinatoric continuum. The smooth components are low-order polynomials, except that 
for M we use an empirical shape that accounts for the phase space limit at M = Ei,. 
The signal and background dependences of JF, Sk, and are bifurcated Gaussian functions. 
We obtain the signal parameters from separate fits to simulated signal, and background 
parameters from fits to the below-threshold data sample. If the simulation estimate of 
background from T(45') production is non-negligible, we add a term with a free fit variable 
to account for this as well. 

Intermediate results for all of the B decay chains appear in Table |. Where relevant, the 
two Ti. hemispheres have been combined. We combine the samples from multiple secondary 
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decay channels by adding the —2 In L functions of branching fraction and extracting a value 
with errors or 90% confidence level (CL) upper limit from the combined distribution. The 
limit is the value of B below which lies 90% of the integral of L. In Table ^ we summarize 
the final results for these measurements with theoretical estimates [|l^]. The first error 
is statistical and the second systematic. The latter include systematic contributions from 
uncertainties in the PDFs 0, reconstruction efficiencies and selection requirements (~10- 
15%). We quote limits computed with efficiencies reduced by one standard deviation. 

We have analyzed each of the decays without use of the likelihood fit, employing more 
restrictive cuts in each of the variables to isolate the signals. The results are consistent with 
those quoted above but with less precision. 



The signals we find in both charge states oi B ^ rjK* are first observations [|TT 



B{B+ r]K*+) = {26Atli ± 3.3) x lO'^ and 8{B^ ^ 7]K*^) = {13.8+Ii ± 1-6) x 10" 
The significance, defined as the number of standard deviations corresponding to the probabil- 
ity for a fiuctuation from zero to our observed yield, is about 5 standard deviations for both. 
We show in Fig. |]the projections of event distributions onto the M axis. A cut has been made 
to reject events with small values of signal £, where for these purposes £ is calculated with M 
excluded. The signals appear as peaks at the B meson mass of 5.28 GeV in these plots. We 
also improve our previous measurements of i? — ^ rj'K with the full CLEO II/II.V data 
sample: B{B+ 7]'K+) = (80^^° ± 7) x lO'^ and B{B° r]'K^) = {89^11 ± 9) x 10-^. 
The M projections for these modes are also shown in Fig. ^. 

Assuming equal decay rates of charged and neutral B mesons we combine 

the measured branching fractions 0. We obtain B(B v'^) = (83lg ± 7) x 10^^ and 
B{B r]K*) = (18.0lti±1.8) x 10"^ We determine 90% CL upper limits for B{B r]'K*) 
and B{B rjK) to be 22 x 10~^ and 5.2 x 10~^, respectively, corresponding to central values 
of (9.0i5;o) X 10^^ and (1.4l^;|) x 10^^ with statistical and systematic errors combined. The 
pattern Bij^K) < BirfK*) < B{r]'K) and Biji'K*) < Biji'K) is evident. 

The observed branching fractions for B rj'K and B — > rjK*, in combination with the 
upper limits for the other modes in Table || and with recent measurements of i? — > Kn, tttt 
1^, i? — > un, pir |jl5|, and CP asymmetry in — ^ Kn, rj'K, ujtt [T^ provide important 



constraints on the theoretical picture for these charmless hadronic decays. The effective 
Hamiltonian calculations [^,^|13| commonly used to account for the charmless hadronic B 
decays contain many uncertainties including form factors, light quark masses, CKM |T7| 



angles and the QCD scale. A large ratio of B{B — >■ rj'K^rjK*) to B{B — > rjK^rj'K*), consis- 
tent with our measurements, was predicted qualitatively [T^ in terms of interference of the 
two penguin diagrams in Fig. [l|(a) and (b), constructive for rj'K and riK* and destructive 
for r]K and rj'K*. Most detailed calculations P,[5|,[T3[| predict a large branching fraction for 
the B — >■ rj'K modes (though usually smaller than the observed values), but no enhance- 
ment of 5 — rjK* . Three recent analyses [1T9H21[], all of which take guidance from charmless 
hadronic B decay data, show that the expectations for B — >■ rjK* can easily be enhanced; the 
effective Hamiltonian calculations accomplish this by increasing the relevant form factor or 
decreasing the strange quark mass, the latter in accordance with recent lattice calculations 



2^ . These and previous calculations fall somewhat short of explaining the large rate for 
B — r]'K, suggesting that the solution may involve contributions that are unique to the rj' 
meson. 

We thank George Hou and Hai-Yang Cheng for many useful discussions. We gratefully ac- 
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TABLE I. Intermediate results for final states listed in the first column, with the subscripts 
denoting secondary decays, including 77' r/vr+Tr" [t^tttt) with rj ^ ■yj (77), rj' (p7), and 

rj — > 7r"'"7r~7r° (3vr). The remaining columns give event yield from the fit, reconstruction efficiency e, 
total efficiency with secondary branching fractions Bg, and the resulting B decay branching fraction 
B, with statistical error only. 



Final state 


Fit events 


6(%) 


€Bs{%) 


H(10-6) 


n' K+ 


6.4 


27 


4.7 


88+}^ 


v' K+ 


61+}?. 
" — 10 


29 


8.7 


72+j3 

' —12 


rl 


q 9+3-6 


24 


1.4 


67+26 

" ' —21 


' PI 


29.6+^0 


28 


2.9 


105+i 


'/ rj-Kn" 


O.O+nn 
—0.0 


28 


4.7 


0.0+^-^ 

—0.0 


'/ (Ti " 


4 4+H 


30 


9.0 


5 1+f-? 


r?' 7r° 

'/ rjTrn" 


'-'•^-0.0 


23 


4.0 


o+i-^ 




8+^-i 
" 0.8 


27 


8.3 


1 0+^2 
-^•"—1.0 


n' K*+ n 


O.O+nn 
—0.0 


14 


0.8 


0+30 

"-0 




n 1+3-3 


y 


U.y 


1+39 
1-1 


'/ rjirn-^^ K^^-7T+ 


0.0;°:^ 


16 


0.6 


.^^1 n 

n ' 
'-'-0 


V'fTf ^KOtt + 


r> <-.4-2 Q 


19 


1.3 








20 


2.3 


1 1 +12 
11_7 




o-otfi;^ 


21 


4.1 


7 

n ' 






15 


2.5 


-'--'--6 




q o-i-D. / 

"3-^-3.2 


9 


2.7 


-1 O + ZD 

-■-^-12 




0+0.9 

0-0-0.0 


17 


2.9 


0-OTo.o 




9 9+4.3 
^•^-2.2 


1 7 


5 1 


4 4+?-I 




'-'•'^-4.6 


45 


17.5 


3 5+3-^ 




0+2-0 


29 


6.6 


o+II 

^"-0.0 




0+H-^ 

'-'•'-'— 0.0 


38 


5.1 


+ nn 




0+S-^ 

"^—0.0 


25 


1.9 


"^—0.0 




5 7+H 


46 


18.2 


q 9+3.3 


fin TT"'" 


0+i-i 


30 


6.8 


0+i-I 

'-'•'^-0.0 




0+il-° 


35 


13.7 


0+°-^ 


r?37r7r° 


o.olJ:^ 


20 


4.6 


0.0^^:^ 




9.3i|:i 


22 


2.8 


341}1 


3.6li-J 


15 


1.1 


09+28 
^^^-20 


3.3ti:? 


25 


2.2 






3.ot?:9^ 


17 


0.9 


34t^o 






32 


8.3 


9.7;|« 




s.olt^ 


21 


3.3 


25t}f 




U-U-O.O 


22 


8.5 


0.0t^+ 


mnP'^ 


r. n+4.6 


15 


3.4 




ViiP^ 


9 n+3-2 

^•U_2,4 


26 


10.3 


2.0l|g 




9 q+4.3 
■^•^-2.3 


18 


4.2 


6i^^ 
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TABLE II. Combined branching fractions (Bfn), significance and final result (B). The statistical 
and systematic errors are given for B^t except where the result is not statistically significant, in 
which case they are combined and the final result is quoted as a 90% confidence level upper limit. 
We quote estimates from various theoretical sources [^] for comparison. 



Decay 


mode 




Signif. 

(^) 


B{10-^) 


Theory 
B{10-^) 


B+ ^ 


■ r/'K+ 


SOlg" ± 7 


16.8 


see Bm 


7-65 


B^^ 




89+1^ ± 9 


11.7 


see Bm 


9-59 


B+ ^ 


■ rj'-K^ 


^■^-1.0 


0.0 


< 12 


1-23 


B^^ 




u-u_o.o 


0.0 


< 5.7 


0.1-14 


B+ ^ 


r]'K*+ 


11.11^^0^ 


1.8 


< 35 


1-3.7 


B^^ 


r]'K*^ 




1.8 


< 24 


1-8.0 


B+ ^ 


r]'p+ 


11 9+11.9 


2.4 


< 33 


3-24 


B^^ 


rj'p^ 


"•"-0.0 


0.0 


< 12 


0.1-11 


B+ ^ 


r]K+ 


9 9+2.8 
^•^-2.2 


0.8 


< 6.9 


0.2-5.0 


B^^ 




'-'•"-0.0 


0.0 


< 9.3 
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FIG. 2. Projections onto the variable M. The histograms show (a) i]K*'^; (b) 

rjK*^; (c) B^ rj' K^; (d) B^ rj'K^. In (c) and (d) the shaded histograms correspond 
to the T]' — > T^vrvr, rj ^ decay chain, while the unshaded histograms correspond to the r]' — > p'y 
channel. The solid (dashed) line shows the projection for the full fit (background only) with the 
cut discussed in the text. 
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